So the batching system (#392) let you do parallel inference but, in order to do parallel training, you need to be able to sum up (reduce) what you're learning online across all elements in the batch and apply them to the (shared) weights. This PR implements this via some new VarAccess modes: REDUCE_SUM and REDUCE_MAX which signal that writes to these variables should be reductions. I went backwards and forwards a lot about the syntax for this one but ended up not really adding any new syntax so a gradient reduce and zeroing custom update might look like this:

gradient_batch_reduce_model = genn_model.create_custom_custom_update_class(
    "gradient_batch_reduce",
    var_name_types=[("reducedGradient", "scalar", VarAccess_REDUCE_BATCH_SUM)],
    var_refs=[("gradient", "scalar")],
    update_code="""
    $(reducedGradient) = $(gradient);
    $(gradient) = 0;
    """)

The nice thing with this lack of syntax is that you can do stuff like implement softmax with $(reducedGradient) = exp($(gradient)); (although that doesn't make a lot of sense reducing across batches) and backends which don't support batching (like single-threaded CPU) can basically just stick in a write back to global memory after the generic code generation and this will automatically turn into an (unnecessary) copy operation.

Because typically NUM_BATCHES << NUM_WEIGHTS, this reduction is quite different than those typically talked about in the literature (https://developer.download.nvidia.com/assets/cuda/files/reduction.pdf) so I dug into the TF source to see how they implement reductions of this type and, for NUM_WEIGHTS > 4096, they use this very simple algorithm:

1. Launch kernel with one thread per weight/neuron
2. Initialise register to a suitable initial value (hence the 'type system' wrangling so a sensible initial value for REDUCE_MAX can be established)
3. In each thread loop over the batches and apply the reduction operation
4. Write the reduced value back to memory.

This makes sense as you get good coalescing of global memory reads and no need for atomics etc and, as GeNN will fuse any compatible reductions together so they're run in parallel, I think any reasonable model will easily occupy the GPU (which I think the 4096 vaguely represents).

I've been using this to do parallel eProp where one of these reductions on the gradients is followed by an Adam optimizer custom update which applies the now-non-batched gradients to the shared weights (via #446). However, it's pretty flexible so you could actually use it with STDP or whatever - you'd apply your STDP rule to a deltaG variable which would be duplicated across the batches, reduce these and add them to the (shared) weights.

On the Titan V, increasing the batch size gives decreases the effective time to train a single stimuli by around 4.5x as shown: